{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cb83dd63",
   "metadata": {},
   "source": [
    "# Tutorial\n",
    "## See the effect of Intersectional Fairness (ISF) technology using RejectOptionClassification (ROC) and the AdultDataset\n",
    "In this tutorial, we will detect the intersectional bias of the AdultDataset, improve the intersectional fairness with ISF, and demonstrate its effectiveness.  \n",
    "While ISF supports several mitigation methods, we now select RejectOptionClassification and extend it for intersectional fairness.  \n",
    "We will also compare ISF with ROC to explain ISF is suitable for retaining intersectional fairness."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f7b1448",
   "metadata": {},
   "source": [
    "#### RejectOptionClassification\n",
    "Reject option classification is a postprocessing technique that gives favourable outcomes to unpriviliged groups and unfavourable outcomes to priviliged groups in a confidence band around the decision boundary with the highest uncertainty.\n",
    "\n",
    "References.  \n",
    "F. Kamiran, A. Karim, and X. Zhang, “Decision Theory for Discrimination-Aware Classification,” IEEE International Conference on Data Mining, 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "df066ab4-3146-4e0a-8f52-ec3a222718c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "sys.path.append(os.path.abspath(\"../\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "61f434b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ece30760",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pylab import rcParams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8240dbba-da06-4f16-9a53-3262562dd0d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aif360.algorithms.intersectional_fairness import IntersectionalFairness\n",
    "from aif360.algorithms.isf_helpers.isf_utils.common import output_subgroup_metrics, convert_labels, create_multi_group_label\n",
    "from aif360.algorithms.isf_helpers.isf_analysis.intersectional_bias import plot_intersectionalbias_compare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4fd1a618",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d24d6739",
   "metadata": {},
   "source": [
    "## Set up AdultDataset\n",
    "Note: To download and install the AdultDataset manually, follow [README.md in AIF360](https://github.com/Trusted-AI/AIF360/tree/master/aif360/data).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c6258781",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aif360.datasets import AdultDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "55ccad01",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Missing Data: 3620 rows removed from AdultDataset.\n"
     ]
    }
   ],
   "source": [
    "dataset = AdultDataset()\n",
    "convert_labels(dataset)\n",
    "ds_train, ds_test = dataset.split([0.7])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bc8d7f02",
   "metadata": {},
   "source": [
    "### Ensure what attributes are protected  \n",
    "To verify intersectional bias, you need to specify two attributes in the Dataset as protected ones.  \n",
    "AdultDataset has already specified the following two attributes as protected:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c5c26fd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['race', 'sex']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.protected_attribute_names"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1aaea4b3",
   "metadata": {},
   "source": [
    "## Classification  \n",
    "We first build a classification model using ROC which is a postprocessing algorithm.  \n",
    "We train a Logistic Regression model with data ds_train and then we proceed with classification for data ds_test.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5a9a4b91",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scale_orig = StandardScaler()\n",
    "X_train = scale_orig.fit_transform(ds_train.features)\n",
    "y_train = ds_train.labels.ravel()\n",
    "X_test = scale_orig.transform(ds_test.features)\n",
    "Y_test = ds_test.labels.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "81f3684e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>LogisticRegression</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.linear_model.LogisticRegression.html\">?<span>Documentation for LogisticRegression</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>LogisticRegression()</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "lr = LogisticRegression()\n",
    "lr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "eb262b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_test_classified = ds_test.copy()\n",
    "pos_ind = np.where(lr.classes_ == ds_train.favorable_label)[0][0]\n",
    "\n",
    "ds_test_classified.scores = lr.predict_proba(X_test)[:, pos_ind].reshape(-1, 1)\n",
    "ds_test_classified.labels = lr.predict(X_test).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b1dc4d34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate ds_train_classified for ROC\n",
    "ds_train_classified = ds_train.copy()\n",
    "pos_ind = np.where(lr.classes_ == ds_train.favorable_label)[0][0]\n",
    "\n",
    "ds_train_classified.scores = lr.predict_proba(X_train)[:, pos_ind].reshape(-1, 1)\n",
    "ds_train_classified.labels = lr.predict(X_train).reshape(-1, 1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dc7811c1",
   "metadata": {},
   "source": [
    "### Confirm the model performance  \n",
    "#### (1) Measure the performance for classification\n",
    "Check the performance for classification using the above classification results.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "71f1958d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_acc = pd.DataFrame(columns=['Accuracy','Precision','Recall','F1 score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "39b7d958",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "df_acc.loc['LR Model']=(\n",
    "    accuracy_score(y_true=Y_test, y_pred=ds_test_classified.labels),\n",
    "    precision_score(y_true=Y_test, y_pred=ds_test_classified.labels),\n",
    "    recall_score(y_true=Y_test, y_pred=ds_test_classified.labels),\n",
    "    f1_score(y_true=Y_test, y_pred=ds_test_classified.labels)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "85fcd4c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1 score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LR Model</th>\n",
       "      <td>0.848824</td>\n",
       "      <td>0.744169</td>\n",
       "      <td>0.607855</td>\n",
       "      <td>0.66914</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Accuracy  Precision    Recall  F1 score\n",
       "LR Model  0.848824   0.744169  0.607855   0.66914"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1db4f8d7-da51-4274-8d20-fb009e02eb6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aif360.sklearn.metrics import one_vs_rest, disparate_impact_ratio\n",
    "\n",
    "def calc_intersectionalbias(dataset):\n",
    "    df, _ = dataset.convert_to_dataframe()\n",
    "    y_true = df[\"income-per-year\"]\n",
    "    y_true.index = pd.MultiIndex.from_frame(df[[\"race\", \"sex\"]])\n",
    "    v, k = one_vs_rest(disparate_impact_ratio, y_true, prot_attr=['race', 'sex'], return_groups=True)\n",
    "    df_results = pd.DataFrame({\"Group(race, sex)\": k, \"Disparate Impact\": v})\n",
    "    df_results[\"Disparate Impact\"] = 1 / df_results[\"Disparate Impact\"]\n",
    "    return df_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0007d28c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Group(race, sex)</th>\n",
       "      <th>LR Model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(0.0, 0.0)</td>\n",
       "      <td>0.190748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(0.0, 1.0)</td>\n",
       "      <td>0.892224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(1.0, 0.0)</td>\n",
       "      <td>0.352248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(1.0, 1.0)</td>\n",
       "      <td>2.810223</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Group(race, sex)  LR Model\n",
       "0       (0.0, 0.0)  0.190748\n",
       "1       (0.0, 1.0)  0.892224\n",
       "2       (1.0, 0.0)  0.352248\n",
       "3       (1.0, 1.0)  2.810223"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_lr_di = calc_intersectionalbias(ds_test_classified)\n",
    "df_lr_di = df_lr_di.rename(columns={\"Disparate Impact\": \"LR Model\"})\n",
    "df_lr_di"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3ff75d14",
   "metadata": {},
   "source": [
    "* The model requires bias mitigation because DIs for groups other than race=0.0 and sex=1.0 are out of the range for fairness. \n",
    "\n",
    "   Note:  \n",
    "   In the recruitment field in the US, there is a law saying it is fair if the DI is 0.8 or more (and equal to or less than 1.25, the reciprocal of 0.8), so we consider 0.8 as a standard threshold of fairness.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d220fe7d",
   "metadata": {},
   "source": [
    "## Run ISF\n",
    "Mitigate intersectional bias in this LR model's judgment.  \n",
    "You can use the ROC algorithm, a post-processing method, for it.  Run the mitigation algorithm of ISF specifying \"RejectOptionClassification\" as a parameter.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7ed1a91c-1779-47fa-bf9c-58cb36707f64",
   "metadata": {},
   "outputs": [],
   "source": [
    "ID = IntersectionalFairness('RejectOptionClassification', 'DemographicParity', \n",
    "                             accuracy_metric='F1', options={'metric_ub':0.2, 'metric_lb':-0.2}, max_workers=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2430417e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [32:39<00:00, 326.53s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<aif360.algorithms.intersectional_fairness.IntersectionalFairness at 0x26ba2a1a050>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# training can take several minutes depending on hardware\n",
    "ID.fit(ds_train, dataset_predicted=ds_train_classified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f9c4cb64",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [04:13<00:00, 42.25s/it]\n"
     ]
    }
   ],
   "source": [
    "# predict\n",
    "ds_predicted = ID.predict(ds_test_classified)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8483bcc3",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "16dee42f",
   "metadata": {},
   "source": [
    "### (1) Compare the raw LR model and the model mitigated by ISF  \n",
    "Measure and visualize DIs for intersectional bias to check the effect of ISF.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "23747507-627c-490a-8cc1-243b36daed90",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_intersectionalbias_matrix(dataset):\n",
    "    df = calc_intersectionalbias(dataset)\n",
    "    df[[\"race\", \"sex\"]] = pd.DataFrame(df[\"Group(race, sex)\"].tolist(), index=df.index)\n",
    "    df_pivot = df.pivot(index=\"race\", columns=\"sex\", values=\"Disparate Impact\")\n",
    "    df_pivot.columns = [f\"sex={col}\" for col in df_pivot.columns]\n",
    "    df_pivot.index = [f\"race={int(idx)}\" for idx in df_pivot.index]\n",
    "    return df_pivot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "245b5005-97e7-4682-a344-3647c9c2f859",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex=0.0</th>\n",
       "      <th>sex=1.0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>race=0</th>\n",
       "      <td>0.190748</td>\n",
       "      <td>0.892224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race=1</th>\n",
       "      <td>0.352248</td>\n",
       "      <td>2.810223</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         sex=0.0   sex=1.0\n",
       "race=0  0.190748  0.892224\n",
       "race=1  0.352248  2.810223"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_intersectionalbias_matrix(ds_test_classified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6a5b15b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgAAAADVCAYAAAAhFEhPAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAALJdJREFUeJzt3Ql4TOf+B/DvxB6hRCyhVPkLqrY2RUhRtGksjfU2KVKttUQV1ZYulqtaRXtvN01iqa260CBaS8WWKyUliPQK1SppuAnBFWpL5P/83tyZGpLIcGY5c76f55knM3NOZs7MM79zfu/vfc97THl5eXkgIiIiQ/Fw9gYQERGR4zEBICIiMiAmAERERAbEBICIiMiAmAAQEREZEBMAIiIiA2ICQEREZEBMAIiIiAyICQAREZEBlXT2BhjRH3/8gT59+mDXrl23LHvttdeQkJCAypUrq8c5OTm4fv06XnnlFTz22GO3rN+wYUM0atQIq1evtnr++eefx88//1zgexRmy5YtWLBgAZYsWVLoOvJ6M2fOxLffflvs1yXSm06dOuGzzz6Dn58fMjIy8Pbbb+Po0aNqWaVKlTBhwgQ0a9bMEoOynofHX+2pHj16YMiQIep+dHQ07rvvPjzwwAN44okn0KBBA8t62dnZaNGihXr9cuXKqee2bt2q3vvs2bMoVaoU6tevj/Hjx6NOnTqW/0tMTMQnn3yCc+fOITc3V23DxIkT4ePjY/U5Dhw4gM8//xxz5syBs7Vu3RorV65Un/Ott97CRx99ZPWdkeMxAXBBw4YNw4ABAyyPN27ciDfeeAM7duwocP3//ve/+PXXX9WOQpw6dQq//PKLw7aXyJ29+eabKiH48MMP1eMff/xRxagcqMuWLaue+/LLL1G+fPlb/vfYsWOIj4/H0KFDVeJfoUIFq2T96tWrCAsLw6pVq9Tf77//Xr3P+++/rxIGsW7dOjzzzDP46quvUKtWLXXwl4bC3Llz1YFfZnOX+4MHD0ZMTIzVQbVp06YucfC/UZUqVdC8eXMsX74c/fv3d/bmGBrTLx2QHcc999xT6PJu3bohNjbW8vi7777Dk08+abWTkQRC1uvevTv+8Y9/qJ2GWLZsmWqVSEVC/s9MWhbjxo1D7969VWtm/vz5dvt8RK4sMzMTly9ftsRMQECAqoIV5zIqUVFR6NWrV6HLJXm/cOGCJb4/+OADlXCYD/4iODhYxW5kZKR6LC3nUaNGqYO/MJlMGDFihEoAJNZvrthJDAtJHCTO5XG/fv2we/fuW7ZHXls+m9mLL75oqfbJtsm+QP5fnr906ZJ6fv369ep1e/bsqSqPkvQIaYTI+zz11FMqYZFKhZl8J/PmzbN6jhyPCYALkp1GSEiIKvkHBgbi8OHDKsMvjASltBzM5L4c6M0+/vhjXLlyRSUJUoLbs2cPvvnmGxw8eFC9l7Qsvv76a7WO2YwZM9R7S/CvWLECmzdvxrZt2+z4qYlc08svv6ziROJBDnySND/00EOWkr0IDQ1VMSs3uS8kQdi0aRM6dOhgVfKXdeSALonECy+8gPDwcHTt2hVnzpzB8ePH1WvfrE2bNti3b5+6n5KSoroNbiStfjnQmisSBZEuA3k/iWlJ7n/66adifwcnT55U+wmpMMj/S5eG7Jd+++039d0sWrRIVTHks8j3JaSbZNCgQVizZg26dOmiPrtZ1apV4e3tjeTk5GJvA2mPXQAu3AWQlZWF5557TvX9ScAVpnbt2mrMgASTl5eXak1Imc1Mug6kf1B2EmXKlEHfvn3VjklaHu3bt7eMN5DnJSsX27dvVwmCBLb4888/cejQIVW6IzISOfBL8iuJs7SiJXmWg54cCM1xVlAXgPThS4tcDnRmN3YBSAlcxgfIwdHckhfXrl2zSi7EjS17We9OWs5S6ZNKoMS+fCZprRdXtWrVULduXdX679ixo9pm2RdIMnTixAmrUr4kMqdPn8aRI0dUYiNk/Ru/ByH7NRlX0bJlS5s/C2mDFQAXJjuXWbNm4dNPP1U7n6JIFUBa+HKTlsCNbi5VymPZyciORAYYmpUs+Vc+KM9L1UF2VnKT7H/gwIGafTYiPZCD2d///ncVG9IKlwqAtIJ9fX3VGICi3BxfN5M+fzmITpo0ST2WRFwOsgWV5qW1bm71y//s37//lpgePXq0qiAURg7Ssu1ywJVWeUH977LNN+4vzIlHiRIlVMIyZcoUlC5dWg1KlIO/JCKSEJj3E+aKoXlfcuNr3bh/EfK/HAToXPz2XZz080mgyijhonYmkmnHxcWpUr25RWHWrl07dQCX/5cyvwSp7Mzatm2rWvqSrUugyk7BTMqTixcvVvfPnz+Pp59+Gjt37rTjJyVyPRUrVlSD+L744gvLwUxa9lKdk7NviiIHdCnJy6DcwkjfeFJSkmqVm8vm0v3273//27LO2rVr1fgcGUhorhBKOV8qcuYDqXTzpaWl4d577y30vaRr4vfff1exPHnyZNVCv7mSINucmpqq7svZD+ZuB3lO+vgbN26MiIgIdV/eX0b2y9lD8t5CvifpZpAzJWTfZa52yMBJGUtx89im+++/v8jvkOyLXQBOIoPsbi59FXbKngz4efzxx1Vm/be//a3AdaS8JmcBSADfXD4cOXKkSiCkMiAtfxlbIF0MkpGPGTNGJRhSvrxx4JGcpjN16lQ1lkD+Rwb5yP/Zclohkd5JjEi3mAyMk4GwElvSjSZ96E2aNLnt/0syLl1wcsAsSPXq1dXpgu+++67qjpP15T0kCZDqgxyg5RRDObBKV585OZf4lMGCktBLK11G+8t2FtWilm2W/YC08mW96dOnq5b9zZVEaUQEBQWpEr0c4IUkO9LSl88h+wpJjGQba9asiddff13tY8xdHOazDuSvdD0uXLhQfYYaNWpY3kc+m+wDzadSknOY8oozlJWIiGwmLW7pd1+6dKmzN8WlSPeiJBE8DdC52AVARGQn0qcvXXAbNmxw9qa4DOlylDEM5rMlyHlYASAiIjIgVgCIiIgMiAkAkU7IKVxytocM0DKPGr+RnDIqE8zIQC6ZnY2I9C/GjnHPswCIdEBOyZKpYOVMEPP88XIqp0z8JOR0LTl/XE7llNPUZGcgI8pvPveaiPQjw85xzwoAkQ7IFSJl9jYJfDnl09/fX52fbiZzPMgOQq4eKfPWy1XkiEjfEuwc92weEDlY586di1wuEzrdTCZRkelYzeSyrzdOMCPzx8ukKnIuuUzbLOdms/VPpN+Yd0Tcu8Qe4sc6+XNgk3baDnb2FriXvMnFP1nGNLXo33MndLL9/fPyLHPFm9046YtcMloutiIzO8okKzLRk5x+Zp48xhXd7nsi0kvcm+wQ846Ie5dIAIjciek2+4zCsv2iSCvAPEWrkKlob5yJTkqFctlYmaVO5qlv1aoVDhw44NIJAJG7MNkh5h0R9xwDQKQxj7yib3dCpn+Vvj/J9mUKVbkqnfQHmslUq3JxGmkxyFUeZXDQ7eaqJyLXjXlHxD0rAEQas0dhW7J7mTNeZk+TAT9yQRYZFCTXlpdL08o1IuT67NIakIFAcqnXevXq2WFLiOhm9urMsnfcu8RMgBwDoD2OAXDeGIDSbxX9e746zekh5xI4BoDcJe5L6zTmWQEg0tjdlPyISH/0GvNMAIg0xnYtkbGYoE9MAIgcPCKYiNyLSacxzwSASGN6bQ0QkbFingkAkcb02h9IRMaKeSYARBrTazmQiIwV80wAiDSm13IgERkr5pkAEGlMr+VAIjJWzDMBINKYXsuBRGSsmGcCQKQxvZYDichYMc8EgEhjei0HEpGxYp4JAJHG9FoOJCJjxTwTACKN6bUcSETGinkmAEQa02s5kIiMFfMezt4AIiIicjxWAIg0ptfWABEZK+aZABBpTK87AyIyVsyzC4BIJ2JiYtC1a1cEBQVh06ZNVsvWrFmDkJAQy61FixaYP3++07aViFw/7lkBINJBayAjIwORkZFYsWIFrl69irCwMLRp0wZeXl5q+VNPPaVuYu/evZg8eTL69++v/YYQkcMqAPaOe5sTgJSUFJw6dQoeHh6oVq0aGjZsqO4TUfHOCe7cuXORy+Pi4m55LiEhAYGBgZbA9/f3R3x8PIKDg63Wu379OqZNm4Y33ngDZcuWhRYY80SOj3lHxH2xE4B9+/bh9ddfR/Xq1VGlShX1XFZWFtLT0zFlyhQEBAQU+02J3Jk9zgnOzMxUB18zHx8fdVC+2ZYtW1C5cmW0atXqrt+TMU/k3HkA7B33xU4AJLP47LPPULt2bavn09LSMHLkSMTGxtr0xkRGLQcWlu0XJS8vDyaT9W6moFb4N998g/DwcGiBMU/kvJh3RNwXu46Xm5uLWrVq3fJ8jRo11EYS0V/lwKJud0JaAadPn7Y8lpb4jS0DceXKFVWub9euHbTAmCdyXsw7Iu6LXQHo2bMnBgwYoEYiVq1a1bIx69evR48ePWx+YyJ3ZY9yoJTbo6OjkZ2drQ7MiYmJGDdunNU6hw4dUv3zpUqV0uQ9GfNEzu0CsHfcFzsBGD58uNqY7du349dff1XPSSYyYcIEdeoBEdlvRLCvry+GDBmC0NBQ5OTkICIiAt7e3urUn6ioKNVP/8cff6jWuVYY80TOPQvA3nFvyrOxlpeamopGjRpZ/mrhxzp6vZSC62o72Nlb4F7yJhc/TBqNLvr3nPqRvsrn9oh5YZrKuCf3iPtGOo15m8/lmThxotVfIrJmus1NbxjzRO4Z83c8ERAHARG517Sgt8OYJ3KvmOdMgEQau5tRv0SkPyadxjyn8/qfyk+EoPkPB9Bi22HcP2MuTCULzo08GzVFyx1HrZ6rMXgMWmxJRbP1+1BvZhQ8ymgzA5uehTQMwYEXDuBwxGHM7TYXJT2sv88yJcpgSa8lap2UF1Iw/OHhlmWhD4bi55E/Y9/wffim3zeoUi5/Ehq90Gs50Gju5jcaVD8Ie4fvVb/RDQM2oGaFmk74BK6vlEcp/DDwB3Rr0O2WZfdXul8tk+8weUQywh4Mg16ZdBrzNicAfn5+6q+cduAuSlWtjnoz5uJgeDD2dfBDCU8v1Hg2wnolkwnVw0ei8dINKFE+f1pGUbHtY/B9bgxS+j6K5Cdb4Mofx1B7wnQYWfXy1dUONXhZMPw+9oNXaS9EtLL+Poc9PEztYJvObYp2C9phasepqFe5Hhp4N8BHwR8h5MsQtIhsgVWpqxDZPRJ6Yq9zgp3FHWP+bn6jYmnvpRjw7QD1G912bBve6fyOkz6J62pZoyV2PL8DbWu3LXB5dI9oLDuwTH2HQUuD8H7Q+6hzTx3okUmnMW9zAjBz5kx1PuLo0aPVYzk1Qe/uefRxZO/egasn/1CPM5dHwyfEOhstW88PFR4KwKHhfayeL//gQzi3bT1ysvKnZzyzIQbewdbrGM3j9R/HjrQd+ON8/vcZnRR9S3bvYfJQO11pdZUrVQ55yMO13GtoVr0Zfkr/CUfOHFHrxaTGoLtfd9WS0Au9tgaMFPN38xs1waSWVSpbSa0n61zKueSUz+HKRj4yEpO3TkZiemKByxftX4SvUr5S909eOImsP7Nwb8V7oUcmnca8zQnA7t271aUJZdpBmae4Y8eO2LVrF/SsdI1aloO/uPqfdJT2tf4hXv71EI68NBDXMk5YPX9x/0+o1P4JlKruqx779OyP0tXy7xtVrQq1LDtWkX4+/ZbAnrt7rtqBnhh3AkfHHMX8vfORdj4Ne/+zFw/5PoRGPvmnm4U3D0eZkmVQxbOKrgYEFXXTG3eM+bv5jUoiMDR2KOLC45A+Lh3hzcIxI36GEz6Fa5PvaN2RdYUuX5K8xJI4DWoxSCVSe07sceAWakevMW9zAvDee+9h8eLFqFixopoURK49PGOGvn/8JtOtX0Pe9evF+t/zu7Yj/bP30Pjz7/Hgqh9V8pB37SqMTFpHN7ueZ/19Sjn1QOYB+M7xRe0Paqs+wt6Ne+O3s79hxHcjsLjnYvw09CdUKF0Bp/88jau5+vlO9VoONFLM381vtIZXDcx5Yg78o/1R6/1amLJtCtaErnHg1ruXUY+MUl0o0u13JfcK9Mik05i3OQGQUqDMPmQm/YJyKUI9u3IyDaVuaLWXql7TqiJQFI/yXvjvvzYhObglUnoG4MLenbj8e3752qikleTr9df3KQOkbmxtiR5+PbBw30Lk5uWqA/yXKV/isbqPoXSJ0jh46iBazWuFR6IfUa0uKbmeuXQGeqHXcqCRYv5ufqOBdQJx6PQhpGSmqPXmJc2DXxU/3Q1WdQX/fPKfauxF4IJA7M/YD70y6TTmbU4A6tati5iYGLVTOHbsmOofrFcvf2CMXv13+0ZUbPUoStfKH4BSLXQwzm4q3pXOyvjeiweWbYJHOU+5TBNqjZqIU6uWwcg2/roRj973qGVAz+CWgxF72Pr7TDqZhH4P9FP3ZaDVk//3pOorLFuyLP71/L/UIC3xRvs3sDxlOfREr+VAI8X83fxGkzOS0aJGC9StVFctk+dPZJ9A1qUsJ3wS/Xq709vwr+mPNvPa4Nez+VNN65WHTmPe5gRg6tSp+OWXX9SFB8aOHat2CtOn63vU+7XTmfht0gg0WhCrTueTyy+eiJqN6gOGo/a4qUX+76UjqfjP5x+h6drdaLH5IK4cP4qT8z6AkWVezMSItSMQGxaL1FH53+fshNnqNCopq4qxG8aqPteDow5i97Dd2H5su+oTPH/lPEavG636V+V/K5etjFd+eAV6otdyoJFi/m5+o4ezDmPM+jHqf+UUttfavYaeX/V09kfSBflu5Tuu6lkVE9pOUN0pWwdtVadUyq1j3Y7QI5NOY97mawFcvnwZ3333Hfr06YOMjAysWrUKgwYNQpkyZe54I3gtAO3xWgDOuxZA62FF/553RbnwHsFBMS94LQByl7hvrdOYt7kCMH78eBw/flzdL1++PC5evKiuDkZE+i4HFoYxT+SeMW9zApCWlqbKgMLLy0tdm9i8cyAi/ZYDC8OYJ3LPmLc5AZB+wKSkJMvj5ORklC3LqW+J9D4iuDCMeSL3jHmbLwY0ZcoUdVnQP//8Uz2uUKGCGhVMRPlcueR3JxjzRO4Z8zYnAE2bNsXatWtx9uxZlCxZUu0MiOgvrlzyuxOMeSL3jHmbE4A9e/Zg3rx5qjUgJxDIhCAnTpzA5s2b7bOFRDpjr5KfnIsfHR2tTsOTQXhdunSxWi4HaZmlT2JzyJAh6Ncv/xz2u8WYJyqaPcv89ox7m8cATJo0CT179oSHhwcGDx6sJgTp1KmTrS9D5LbsMSBITr+LjIzE119/jeXLl2PWrFm4cOGCZfnRo0fxwQcfYOHChVixYgU++eQT1WLXAmOeyDmDAO0d9zZXAOQ0oKCgIBw5ckTtEKZNm4bevXvb+jJEhu0P7Ny5c5HL4+LibnkuISEBgYGBahS+8Pf3R3x8PIKDgy3/06tXL1SqlH+FOpm739PTE1pgzBM5PuYdEfc2VwDkxWUUsMwHvm3bNpWNZGdn2/oyRG7LHiOC5Sp8ciEeMx8fH5w6lX8JavOpenJ+vlyxLyQkRI3av9uJeswY80TOOQvA3nFvcwVAXnzNmjVqVPDKlStVKXDYsGG2vgyR27pdyW9TIdl+UaTvXaarvZG0xs1ycnKQmJiIzz//HNeuXUNoaCiaNWumyZz9jHkix8e8I+Le5gTgzJkzmDt3LkqUKKH+EpH9TwmSVkBqaqrlcVZWFpo0aWLVMpBSoXmE/sMPP6zW1yIBYMwTOec0QHvHvc1dAKVLl1YtgOeeew4jRoyw3IjIfuXAgIAA1fcnpfdz586prF/6A83at2+vyvOXLl1SJcF9+/apkr0WGPNEzukCsHfcl7yTecGJyLHnBPv6+qpTfKTEJ2W/iIgIeHt7q36/qKgolfn37dtX3WT5gAEDUL9+fU3emzFP5Jx5AOwd9zZfDdAeeDVA7fFqgM67GmDXAUX/nr9f6vSQcwm8GiC5S9x31WnM21wBIKKi8bBGZCwm6BMTACKN6XVaUCIyVswzASDSmF4vDEJExop5m88CICIiIv1jBYBIY3ptDRCRsWKeCQCRxvS6MyAiY8U8uwCIiIgMiBUAIo3ptTVARMaKeSYARBrT6znBRGSsmGcCQKQxvZ4TTETGinkmAEQa02s5kIiMFfNMAIg0ptdyIBEZK+aZABBpTK/lQCIyVswzASDSmF7LgURkrJhnAkCkMb2WA4nIWDHPBIBIY3otBxKRsWKeCQCRTsqBMTExiI6ORm5uLiZMmIAuXbpYLQ8PD8fZs2fh4ZE/weeCBQtQpUoV+2wMETmkC8Cecc8EgEgH5cCMjAxERkZixYoVuHr1KsLCwtCmTRt4eXmp5Xl5eUhPT8emTZtgMum1IEmkTyY7va69457XAiCyQzmwqNudSEhIQGBgoAp8b29v+Pv7Iz4+3rL8t99+Uy2EZ599Fr169cLGjRu1+0BE5PCYd0TcswJApLHb5eGdO3cucnlcXNwtz2VmZqJatWqWxz4+Pjh16pTl8fnz51XLYMqUKaoc2L9/fzRu3Bi1a9e+g09ARM6OeUfEvUskAEm+zt4CItfuD5RS380lPnOfn2jZsqW6CV9fX3Tq1Ak7d+5kAkCk4zEAeXaOe5dIAIjcye1KfoVl+0WRVkBqaqrlcVZWFpo0aWJ5vGfPHly7dk21BsxKlmR4E+k15h0R9xwDQKQx021udyIgIED1/WVnZ+PcuXNITExU/YFmFy5cwJw5c9RAoTNnzmDr1q1o27atZp+JiBwb846IezYRiHRQDpTy3pAhQxAaGoqcnBxERESoQUEhISGIiopChw4dkJSUhJ49e+L69esYN24cqlevrv2GEJHDugDsHfemPOlkcLJPWvO0Ja1FdHX2FriXvMnFD5OR3Yv+PX+61ukh5xJMUxn35B5xP1KnMc8KAJHGeFgjMhYT9IkJAJHG9HphECIyVswzASDSmF7nBSciY8U8EwAijem1HEhExop5JgBEGtNrOZCIjBXznAeAiIjIgFgBINKYXlsDRGSsmGcCQKQxve4MiMhYMc8uACIiIgNiBYBIY3ptDRCRsWKeCQCRxvS6MyAiY8U8uwCIiIgMiBUAIo3ptTVARMaKeSYARBrT67SgRGSsmGcCQKQxvU4LSkTGinkmAEQa02s5kIiMFfNMAIg0ptdyIBEZK+Z5FgCRHcqBRd3uVExMDLp27YqgoCBs2rSp0PVefPFFzJ8//y7eiYhcIebtHfesABDpoByYkZGByMhIrFixAlevXkVYWBjatGkDLy8vq/XWrFmDnTt3onnz5tpvBBE5tAvA3nHPBIDIweXAzp07F7k8Li7ulucSEhIQGBhoCXx/f3/Ex8cjODjYamfx5ZdfIjQ09E43nYhcJOYdEffsAiDSQTkwMzMT1apVszz28fHBqVOnrNaZPHkyJk2ahFKlSt3lJyAiV+gCsHfcF7sC0K9fP1y+fLnQ5bGxsTa/OZERy4GFZftFycvLg8lkvSvx8Pgrf1++fDkaN26MBx98EFu2bIEWGPNEzot5R8R9sROAt99+G0OHDsXs2bNRs2ZNm9+IyCjsMSJYWgGpqamWx1lZWWjSpInl8caNG3H69Gls3rxZ/ZWdRMWKFdVB/E4x5omcexaAveO+2AmAn58fXn31VSxcuBCffvqprZ+DyDDsMSlIQEAAoqOjkZ2djdzcXCQmJmLcuHGW5RKXZh999BE8PT3v6uAvGPNEzp0IyN5xb9MgQDkVQW5E5NjWgK+vL4YMGaIG+uTk5CAiIgLe3t4ICQlBVFQUqlevrv2bMuaJnFoBsHfcm/Kkk8EGUo5o1KiR5a8WPmmt14kUXVcE99mayptc/DCZ3a7o3/PLO/Q1a4g9Yl6YpjLuyT3ifrZOY97mswAmTpxo9ZeIHDcpiDMw5oncM+bveB4AGwsHRIah12lBb4cxT+ReMc+JgIg0ptcLgxCRsWKeEwH9z/3tQxD6xQH0X3EYHV6dC48S1rlRmYqV0XXWKoQuS1Y3vyf7W5a1fXEWBsYcxdNL9qpb29HvOeETuK6X2ryElBdSsH/Efqz820p4l/O2Wl6lXBXEhsWq5XLr1agX9Eyv5UCjCWkYggMvHMDhiMOY220uSnrcFPMlymBJryVqHfn9Dn94uGVZUP0g7B2+F/uG78OGARtQswJPkyxIKY9S+GHgD+jWoFuh63Sp1wU7B++Enpl0GvMed3JqkGjYsCHchad3dXXQX/tSMJb19UOpcl5o2i/Cap1Ww6bhzG8/48v+zbB6dBe0f/ljlK3ko5bVeDAA617rja8GtlS3hI9ecdIncT2BdQIxuOVgtJ7XGs0/a45DWYfwbpd3rdaZ9OgkpJ5OVct7LO+BRT0XoVzJctBzObCom964Y8xXL19dHfSDlwXD72M/eJX2QkQr65gf9vAwlQQ0ndsU7Ra0w9SOU1Gvcj21bGnvpRjw7QC0iGyBbce24Z3O7zjpk7iuljVaYsfzO9C2dttCk4O3OryFr/p+hRIeJaBnJp3GvM0JwMyZM9X5iKNHj1aP5dQEvbu39eM4mbwDFzL/UI//vToaDZ4Is1onfvZo7Ip6S933qnYvrudcQ+61K/AoWQo+fi3g/9wbeHrpfnR+63OUqVDJKZ/DFZ3+8zRGfT8KF69dVI+TTiah7j11rdbxMHmgYpmK6n75UuVxJfcK9EyvrQEjxfzj9R/HjrQd+ON8fsxHJ0Uj7MGwW36XkhhIZaBcqXLIQx6u5V6DCSa1rFLZ/DiXdS7lXHLK53BlIx8ZiclbJyMxPbHA5Y/e9yiqelbF86ufh96ZdBrzNicAu3fvVucFh4eHq3mKO3bsiF27dkHPvKrWwsX/HfzFxVPp6iB/s7zcXHSdtRp95u/Ez6uicO1iNjyr1ED6ni3Y8c/x+GpgC1w+fwYdJ0Y5+BO4LmnZbz+2Xd2vULoC3mz/Jr5N/dZqnenbp6Nj3Y44Me4E9o3Yh0lxk3S9Q5X+wKJueuOOMV+rQi3LwV+kn0/HvRWtY37u7rnqIC+/y6NjjmL+3vlIO5+mEoGhsUMRFx6H9HHpCG8WjhnxM5zwKVybfEfrjqwrdPnmo5sxet1onLt8DnrnodOYtzkBeO+997B48WI13aBMUyjXH54xQ98/fpPp1q8hL+96get+PyEEi7rXQr2OvdW4gQsZafhufA9kn/xd/gm7F/wdddt1lxd1wJbrh/SRbnl2C3b+sROf7f7MatnHXT/G0uSlqPl+TTT8uKHqEnik5iPQK72WA40U89KCv9n1m2JeSv4HMg/Ad44van9QW/Vj927cGzW8amDOE3PgH+2PWu/XwpRtU7AmdI0Dt55cjUmnMW9zAiClwBtnH5J+wevXCz5Y6kV2Zho8q/haHnv61LR0B5jVbv04ylWuqu5fOnsKxxK+U6V/73pN4Bf0zA1rmpB3PVclA5SvWfVmapDPqkOrMHztXwOpzHr49VAlWPH7ud+x/sh6tL+vPfRKr+VAI8W8tOR9vXytEtQbKwLm3+XCfQuRm5erurK+TPkSj9V9TI1rOXT6EFIyU9R685Lmwa+KnxrMSsZk0mnM25wA1K1bFzExMWqncOzYMdU/WK9e/sAYvUrbtRE1WzyKCjXqqMcPPDUYv8dbX+ns/7o8jZYD8wf3SR//fQHBSE/aqg72geP+ifLVaqllLfuPx2/bYpzwKVyTtJZkFPCEHyaoUn9BZFxAvwfy56++p8w96HBfh0L7DfVAr60BI8X8xl83qj7oOvfkx7wMVI09HFvo71IGAz75f0+q32VyRjJa1GiBupXyx7LI8yeyTyDrUpYTPgm5ApNOY97meQCmTp2qLgwi1x4eO3Ys/P39MX16wTt2vbh0JhNbZ45A19mxKFG6DDJ+3oW9y2ajSa/hKF+1JhKjJiPhowl4bFK0OgVQJkTZ/9U/cSJpm/r/f/1jLHr8Y70aEChnCmx5e7CzP5LLmBg4UQ2Sei3wNXUT0nqSswFkpxm5JxLhq8LViGw5zUrKsB8mfoj44/HQK1fu87sT7hjzmRczMWLtCHX6qRzcd6XvwuyE2eo3KNUAGbw2dsNYfNrtUxwcdRA513Pw7cFvsSR5ifr/MevHqP/NvZ6r+rB7ftXT2R9JF6RbxRz37sRDpzFv87UA5Prg3333Hfr06YOMjAysWrUKgwYNQpkyZe54I3gtAO3xWgDOuxZAtH/Rv+ehu/W1t7BHzAteC4DcJe6jdRrzNncBjB8/HsePH1f3y5cvj4sXL2LChAn22DYiXdJrObAwjHki94x5mxOAtLQ0VQYUXl5e6trE5p0DEen3lKDCMOaJ3DPmbU4ApB8wKSnJ8jg5ORlly5bVeruIdEuvI4ILw5gncs+Yt3kQ4JQpU9RlQf/880/1uEKFCmpUMBHlc+WS351gzBO5Z8zbnAA0bdoUa9euxdmzZ1GyZEm1MyCiv7hyye9OMOaJ3DPmbU4A9uzZg3nz5qnWgJxAIBOCnDhxAps3b7bPFhLpjL1KfnIufnR0tDofXwbhdenSxWr57NmzsWXLFnh4eOCFF15Q0/dqgTFPVDR7lvntGfc2jwGYNGkSevbsqd5s8ODBakKQTp062foyRG7LHiOC5fS7yMhIfP3111i+fDlmzZqFCxcuWJbv3LkTKSkpiI2NxZIlS9R5+levXtXk8zDmiZxzFoC9497mCoCcBhQUFIQjR46oHcK0adPQu3dvW1+GyLDlwM6dOxe5PC4u7pbnEhISEBgYqEbhC5mMJz4+HsHBwepxmzZt1HMSk3LBntKlS6NECW0uscqYJ3J8zDsi7m2uAHh6eqpRwDIf+LZt21Q2kp2dbevLEJENJLjlQjxmPj4+OHXqlNU60j//zjvvqINz3759NUsAGPNE7hn3NlcAZPavNWvWqFHBK1euVKXAYcOG2foyRIZtDRSW7RdF+t5NN11hUrL+m0lcSj/gwIEDLa2Du8WYJ3J8zDsi7m1OAM6cOYO5c+eqLEP+EpH9RwRLKyA1NdXyOCsrC02aNLE8Pnr0qOr7k1Z6pUqVVNnw8OHDmiQAjHki55wFYO+4tzkBkD4GaQE0aNDAai7wzz6zvsY7EWknICBAjQSW0ruMBk5MTFQz8pnJzHxRUVFYtGiRmrt/x44dePfddzV5b8Y8kXvGfck7mReciBzbGvD19cWQIUMQGhqKnJwcREREwNvbGyEhIWoH0KFDBzVb31NPPaVa6gMGDMADDzygyXsz5omcUwGwd9zbfDVAe+DVALXHqwE672qAK5sU/Xvu87PTQ84l8GqA5C5xv1KnMW9zBYCI3HNaUCIyVswzASDSmF6nBSUiY8U8EwAijbGwTWQsJugTEwAijem1HEhExop5JgBEGtNrOZCIjBXzTACINKbXciARGSvmmQAQaUyv5UAiMlbMMwEg0phey4FEZKyYZwJApDG9lgOJyFgxzwSASGN6LQcSkbFingkAkcb0Wg4kImPFPBMAIo3ptRxIRMaKeSYARBrTazmQiIwV80wAiDSm19YAERkr5pkAEGlMr/2BRGSsmGcCQKQxvZYDichYMe/h7A0gcsdyYFG3OxUTE4OuXbsiKCgImzZtumX5rFmz0L17d3Tr1g2LFy++q89ARM6PeXvHPSsARDooB2ZkZCAyMhIrVqzA1atXERYWhjZt2sDLy0st37p1Kw4dOoTVq1fjypUr6Nu3L9q1a4f69etrvzFE5JAuAHvHPSsARHYoBxZ1uxMJCQkIDAxUge/t7Q1/f3/Ex8dblvv6+mLs2LEoUaIEPD09UadOHbXzICJ9xrwj4p4VACKN3a7k17lz5yKXx8XF3fJcZmYmqlWrZnns4+ODU6dOWR43bNjQcn///v1ISUlBs2bNbNpuInKdmHdE3LtEAjBql05HULiwUc7eAAMLOH6b3/NtdgYFycvLg8lkvZvx8Li1gLdv3z6MHj1a9Quay4SuKm8y457cQ4AdYt4Rce8SCQCRkRSW7RdFWgGpqamWx1lZWWjSpInVOjt27MCrr76qdgIBAQGabCsROSfmHRH3HANApAMS2NL3l52djXPnziExMVH1B5odP34cr7zyCubOncuDP5GbCLBz3LMCQKQDMthnyJAhCA0NRU5ODiIiItSgoJCQEERFRWHevHlqlPAbb7xh+Z+XX34Zjz76qFO3m4hcN+5NedLJQERERIbCLgAiIiIDYgJARERkQEwAiIiIDIgJABERkQExASAiIjIgJgBEREQGxASAiIjIgJgAEBERGRATACIiIgNiAnAXZBLF6dOnIzg4GL169cLRo0fvaB269RrYI0aMKHCZTHv50ksvoWvXrujfvz9Onz7t8O0jY2Pc2wfj3vGYANyFdevWqWszy9+33nrLaj5mW9ahv3aaS5cuxdixY9X9gixatEjNj/39998jLCwMM2fOdPh2krEx7rXFuHceQyQAJ0+eVFlj79698cwzz+DYsWPYtGmTeiwXVXj99ddx7do1/PDDDypbz83Nxe7du1W2efnyZQwaNEitd+NNLssoV2nq3r27eo+WLVuqgD9z5ozVexdnHb2x1/f5n//8B8nJyZg2bVqh7719+3b06NFD3ZfWlTzm5SyoIIx7bTHu3Y8hrga4cuVKS+lozZo1SEpKwrJly7BkyRKUL19eZZNffPEFnn32WcTGxuLzzz/Ht99+i3feeQdly5ZVjwuSmZmJqlWrWh5XqVJFPSdXa7JlHb2x1/cp3nvvPezatavQ5Td+nyVKlECZMmVw8eJFeHl52eWzkn4x7rXFuHc/hkgA2rRpgxdffBEHDhxAp06d4OHhgbS0NJXFmvuXWrdure5LuU4yzD59+qB58+bqOclcz549a/WaCxYsKDADlde+UXHW0Rt7fZ+yk7wTJpPprj8TuR/GvbYY9+7HEAmAv7+/ykg3b96sstX09HQEBgZizpw5avmFCxcs60o5SrLLgwcPWp4rLHOtVq2a1WAUKWfdmPUXdx29sdf3WRzyfZq/QykxSslRWh9EN2Pca4tx7370nZIW06xZs1RfVb9+/TBmzBiULFlSlZvkRyqZ+ptvvolvvvkGOTk56v77778PT09P9VxR5Me/du1a9Rr79+9HxYoVUblyZZvX0Rt7fZ/FId/n6tWr1f0NGzbgkUce0eATkTti3GuLce9+DFEBkD6r8ePHY/ny5ShVqhRmzJihsvOhQ4eqbPLBBx/EgAEDMH/+fDRo0ACtWrVCzZo11WjTDh06qOyzINIftnfvXjXYR15X+rqElMg+/PBDREdHF7qOntnr+yxMXFycanW8/fbbCA8PV4ONunXrhgoVKqidDFFBGPfaYty7H1Meh1ISEREZjiG6AIiIiMgaEwAiIiIDYgJARERkQEwAiIiIDIgJABERkQExASAiIjIgJgBEREQGxASAiIjIgJgAEBERGRATACIiIhjP/wPKGksDjqonMwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 600x200 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "rcParams['figure.figsize'] = 6,2\n",
    "sns.set(font_scale = 0.7)\n",
    "df_test_classified_pivot = calc_intersectionalbias_matrix(ds_test_classified)\n",
    "df_predicted_pivot = calc_intersectionalbias_matrix(ds_predicted)\n",
    "plot_intersectionalbias_compare(df_test_classified_pivot,\n",
    "                                df_predicted_pivot,\n",
    "                                vmax=0.8, vmin=0, center=1,\n",
    "                                title={\"right\": \"LR Model\", \"left\": \"ISF(ROC is used)\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7162a5fb",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "source": [
    "* Compared to the left-hand quadrant for the LR model with no bias mitigation, DI in each subgroup gets closer to 1.0 in the right-hand quadrant for the model mitigated by ISF.  \n",
    "* This indicates that the judgment result of the model with ISF is closer to fair than that without ISF.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a827905f",
   "metadata": {},
   "source": [
    "Next, ensure ISF does not degrade the accuracy of the model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f3ed6a81",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "df_acc.loc['ISF (ROC is used)']=(accuracy_score(y_true=Y_test, y_pred=ds_predicted.labels),\n",
    "                   precision_score(y_true=Y_test, y_pred=ds_predicted.labels),\n",
    "                   recall_score(y_true=Y_test, y_pred=ds_predicted.labels),\n",
    "                   f1_score(y_true=Y_test, y_pred=ds_predicted.labels) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b1a5aa4d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1 score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LR Model</th>\n",
       "      <td>0.848824</td>\n",
       "      <td>0.744169</td>\n",
       "      <td>0.607855</td>\n",
       "      <td>0.669140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ISF (ROC is used)</th>\n",
       "      <td>0.830692</td>\n",
       "      <td>0.693913</td>\n",
       "      <td>0.584701</td>\n",
       "      <td>0.634643</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Accuracy  Precision    Recall  F1 score\n",
       "LR Model           0.848824   0.744169  0.607855  0.669140\n",
       "ISF (ROC is used)  0.830692   0.693913  0.584701  0.634643"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_acc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "53b11079",
   "metadata": {},
   "source": [
    "* `Accuracy` before and after bias mitigation are almost the same.  \n",
    "* This indicates ISF can mitigate intersectional fairness with only minor accuracy degradation.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "85d3578c",
   "metadata": {},
   "source": [
    "### (2) Comparison of ROC and ISF (ROC is used)\n",
    "Now compare the effects between ROC and ISF-leveraged ROC.  \n",
    "Run ROC under the same condition as the ISF-leveraged ROC.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "b91aa562",
   "metadata": {},
   "outputs": [],
   "source": [
    "from aif360.algorithms.postprocessing.reject_option_classification import RejectOptionClassification\n",
    "        \n",
    "ROC = RejectOptionClassification(\n",
    "            privileged_groups=[{'race':1,'sex':1}],\n",
    "            unprivileged_groups=[{'race':0,'sex':0},{'race':0,'sex':1},{'race':1,'sex':0}],\n",
    "            low_class_thresh=0.01, \n",
    "            high_class_thresh=0.99,\n",
    "            num_class_thresh=100, \n",
    "            num_ROC_margin=50,\n",
    "            metric_name='Statistical parity difference',\n",
    "            metric_ub=0.2,\n",
    "            metric_lb=-0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1ab6402a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# training\n",
    "ROC.fit(ds_train, ds_train_classified)\n",
    "# predict\n",
    "ds_predicted_roc = ROC.predict(ds_test_classified)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d61f78e7",
   "metadata": {},
   "source": [
    "#### Check intersectional bias for ROC  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e12f9ac8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Group(race, sex)</th>\n",
       "      <th>LR Model</th>\n",
       "      <th>ROC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(0.0, 0.0)</td>\n",
       "      <td>0.190748</td>\n",
       "      <td>0.324319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(0.0, 1.0)</td>\n",
       "      <td>0.892224</td>\n",
       "      <td>1.264260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(1.0, 0.0)</td>\n",
       "      <td>0.352248</td>\n",
       "      <td>0.538010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(1.0, 1.0)</td>\n",
       "      <td>2.810223</td>\n",
       "      <td>1.753728</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Group(race, sex)  LR Model       ROC\n",
       "0       (0.0, 0.0)  0.190748  0.324319\n",
       "1       (0.0, 1.0)  0.892224  1.264260\n",
       "2       (1.0, 0.0)  0.352248  0.538010\n",
       "3       (1.0, 1.0)  2.810223  1.753728"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_roc_di =calc_intersectionalbias(ds_predicted_roc)\n",
    "df_roc_di = df_roc_di.rename(columns={\"Disparate Impact\": \"ROC\"})\n",
    "df_lr_di[\"ROC\"]=df_roc_di[\"ROC\"]\n",
    "df_lr_di"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a55eca5b",
   "metadata": {},
   "source": [
    "* Since ROC does not support intersectional bias, DI values for groups are out of the fairness range though they tend to improve."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1ee272aa",
   "metadata": {},
   "source": [
    "#### Compare DI values for `ROC` and `ISF (ROC is used)`\n",
    "Finally, compare DI values ordinal ROC and `ISF (ROC is used)` achieve.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "611ccd2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgAAAADVCAYAAAAhFEhPAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvc2/+5QAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKxxJREFUeJzt3QlYVOX+B/AvCK5IirjgdtVuLlmmN1RQzAW7iKa43iSNLNGs1ERTyyyXSnPL270VCWpplpa7WKmJSyRX+bvrTbRSE5cEURMtF3D+z+/1MjkpyOCZGd4538/zzCNnznE4wzPfM7/zvu95j4fFYrGAiIiITMXT1TtAREREzscCgIiIyIRYABAREZkQCwAiIiITYgFARERkQiwAiIiITIgFABERkQmxACAiIjIhFgBEREQm5OXqHaCCO378OP7+97/jvvvusz6XlZWFxo0b46233kKpUqXy/L/nzp3DzJkzsWvXLnh6eqJkyZIYPHgwWrVq5aS9J3If7dq1w4cffoi6devi9OnTKn9HjhxR68qVK4eRI0eiUaNGarlevXpqO8ldrs6dOyM6Olr9HB8fj7/85S8q244ydOhQtGnTBt27d7d5/uWXX0ZycjLKly+vlrOzs3H9+nWMGjUKbdu2vePrrlu3Dh999BF+++039X+bNm2q3nuZMmXwxRdfwMPDA7169XLY+6K7JFMBkx7S0tIszZo1s3nuypUrlu7du1s+++yzPP/f5cuXLY899phl3rx5lpycHPXcgQMHLEFBQZY9e/Y4fL+J3E3btm0tBw8eVD8PGDDAsnDhQuu65ORkS/PmzS2///67Wq5bt67l4sWLt32do0ePWp588kmH7++QIUMsS5cuveX50aNHWz755BOb59auXWtp0aLFHV9zxYoVls6dO1tOnDihlq9evWp57bXXLM8++6xalmNNz549LRkZGYa9DzIWuwA09+uvv+LixYu455578P333yMyMlKdXfTs2RPbtm1T23z11VdqfVRUlPUspH79+vjnP/+JsmXLuvgdEOktPT0dly9flpMptRwcHIwpU6ZYl/MTFxeHbt26qZ8lrzefoS9YsECdoYuvv/4aXbt2RY8ePfDEE0/gp59+Us+npqbiySefVK8h67799lv1vBwThgwZgvDwcDzzzDP45Zdf7GpplOOFkPcwY8YMdOzYUR1Xxo8fjytXrqh17777LsaMGYOqVauqZW9vb9VyINvJ/5NjTVhYGD7++OMC/25yLnYBaEaa/CMiIlRz29mzZ1GtWjX1xf7oo4+qsE2cOBEhISE4cOAA+vfvj9WrV2P//v146KGHbnmt5s2bu+Q9ELmTl156SX3xSVP+ww8/rHLVpUsXmy653r17W4tveX7RokXqS3L9+vUYMWLEHX+HfAm/9957qnBftWoVdu7ciZo1a6rfLV/E9957r/qSf/zxx7F8+XLVPeHn56cKhxMnTqj9ya8IWbx4MS5cuIBr166p40dsbKxaJ834e/bsUa/p5eWl3ucHH3yAfv36qdeV7seb+fj4oFOnTtbl1q1bq65G2U8qelgAaEbO2FeuXKl+XrhwoTrotG/fHkePHkWxYsVUeEWDBg1Uv+Pu3btVP5z06xGR8SRzmzdvxo4dO5CSkqK+TOVLddmyZahQoYLaRr7wpV/8z+Nyrl69qr6o70SK+wEDBqixBzJuR75kDx8+jGPHjmH48OHW7STr0jrwn//8BxMmTFDPyUlC7nHhdgYOHIi+ffsiMzMTTz/9tCosZEyC2LJli2pZKFGihFqWFsZJkyap7UROTk6++y2vJfso28nxiYoWdgFoTMIoZ/bSDHe7L3g5w5CWAhmMJIXAn0kTo1T4RFQ40gr3xhtvqLPjoKAgNdhOzpYDAgKwadOmfP/vnwtzWb6ZnI3nkoF18+fPV2f6c+fOVV/68qXq7++vTghyH59//jmaNGmi/s/Nry37dydSrEybNk2d4UsxI27XjSH7JQMdpUiQ1oGbXbp0SRUq0lIpZB+l5ePP742KBhYAmpM+QmkOlKpfwvbdd9+p5w8ePIj//ve/qomuQ4cO6mxj3rx51oPC3r17VdBlhDIRFY6vry+SkpLw2WefWb8sJWtyNi3N9fmRkfdyNU5GRoZ1WZrV5ctTvmSle0BIrkNDQ9XP0t03bNgw1fdfp04dtS53OxkDJFmXpnw545cWCNkneX0Z6V8Qcjzo06ePuqpBjhUtW7bE0qVLVb+/LEuBIYWOGDRoECZPnqz2WUhrhrQOyJl+7tiitLQ01Qpw8xUQVHSwC0BzlStXVpcTvfPOO+oyv7ffflsNQJLASb9hpUqV1Hby5S/PS4AloKVLl1bV/u3GBhBRwciZ9ezZs1W25syZo/r3pblcztAbNmx4x/8v3XfSzC4D/OTyXhlAJ2N8JLdyJi/FhORVWgBkUJ8MtJPfKYPxihcvrsYFyJeujAMQchyQLgXZ9vXXX1cFgRwj7Cn0X3jhBTWmaMmSJfjHP/6hBgXK4ERpTfzb3/6mChAhz0lRINtLoSEFQIsWLdQ+5JLCw5GXN9Ld8ZBLAe7yNYiIqBBk7M7YsWNVd5y7keJArkaScUq5YyGoaGG7DBGRi9SqVUs1s69duxbuRlob5aoEfvkXXWwBICIiMiG2ABAREZkQCwAiTcjlZTIjm1wTnjvy+2YyqFOuD5eBZGvWrHHJPhKRPrnnVQBEGpAbzsyaNUuNzJbR1jIHhFyOJTOvCZnnYfv27WqWOBk5LgcDGWFekOu/icicuWcLAJEG5HIqubZbgi+XeQUGBqrrz28ecS0HCLlUS+all8vFiEhvyQ7OPU8PiJwsd1KXvCQmJt72hjO5czoImQEudwIZIddn165dG4888oi6Neurr77Ks38ijTPvjNwXiSPE3iqcJtJoXSJdvQfu5ejMgl8sUzsm/89zHbSz+/fLxTp/nk715tnV5L7sMoOc3A1OpqeVud3l8rIaNWqgqKp1h78TkS65r+2AzDsj90WiACByJx53OGbkVe3nR84CZPrXXDLV7M0zzUlTodz6VWahk3nomzVrhn379hXpAoDIXXg4IPPOyD3HABAZzNOS/6Mw5B7z0vcn1f758+fVXeekPzCX3PlRbj4jZwxyL3gZHHSnueiJqOhm3hm5ZwsAkcEc0bAt1b3c80HuKy8DfuQe6zIoSOaNl1vPypzthw4dUmcDMhDomWeeUTeLISLH83DQ6zo690ViJkCOATAexwC4bgxA3aH5f54P/cvlkSsSOAaA3CX3dTXNPFsAiAx2N01+RKQfXTPPAoDIYLoeDIjIXJlnAUBkMDZsE5mLB/TEAoDIyZcEEZF78dA08ywAiAyma3MgEZkr8ywAiAyma3MgEZkr8ywAiAym69kAEZkr8ywAiAym68GAiMyVeU4FTEREZEJsASAymK5nA0RkrsyzACAymK4HAyIyV+ZZABAZTNdrgonIXJlnAUBkMF3PBojIXJlnAUBkMF2vCSYic2WeBQCRwXRtDiQic2WeBQCRwXRtDiQic2WeBQCRwXQ9GBCRuTLPiYCIHNAfmN+jsJYvX46OHTsiLCwM69evt1m3atUqREREWB+NGzfGnDlz7vq9EJHrMu/o3LMFgEiD/sDTp09j1qxZWLJkCa5evYrIyEgEBQXBx8dHre/SpYt6iF27dmHcuHHo06eP8TtCRE4bA+Do3NtdAOzfvx8ZGRnw9PREpUqVUK9ePfUzERWsOTA0NDTf9YmJibc8l5ycjJCQEGvwAwMDkZSUhPDwcJvtrl+/jokTJ2Ls2LEoWbIkjMDMEzk/887IfYELgN27d+PVV19F5cqVUaFCBfVcZmYmTpw4gfHjxyM4OLjAv5TInd2pya8wJwvp6enqyzeXv7+/+lL+s40bN6J8+fJo1qwZ7hYzT+S6zDsj9wUuAKSy+PDDD1GjRg2b59PS0vD8888jISHBrl9MZNazgW/yqPbzY7FY4OFhe5i53Vn44sWLERUVBSMw80Suy7wzcl/gdrycnBxUq1btluerVKmidpKI/jgY5PcoDDkLOHPmjHVZzsRvPjMQV65cUc31LVu2hBGYeSLXZd4ZuS9wC0DXrl3Rt29fNRKxYsWK1p1Zs2YNOnfubPcvJqKCk+b2+Ph4ZGVlqS/mlJQUDB8+3GabgwcPqv55b29vQ34nM0/k3rkvcAHw7LPPqp359ttv8dNPP6nnpBIZOXKkuvSAiBx3TXBAQACio6PRu3dvZGdnY/DgwfDz81OX/sTFxal++uPHj6uzc6Mw80SunQfA0bn3sNjZlpeamor69etb/zXC3iq6zqRcdHWJdPUeuJejMwsek8598v88J3yqV/O5IzIvasUw9+Qeue+saebtvpbnlVdesfmXiG69Jji/h26YeSL3zHyhJwLiICAi95oW9E6YeSL3yjxnAiQyGBu2iczFA3ridF7/49shAnU37UO95EOoNiUW8LKtjYqVK4+/fLwC923cqx7levwx3WK5bpG4b8Me3Je4G7W/+AbeNWu54B0Ubd7FvLHguW/Q7v5Ot6yr7lcLnz2fiK9f2o0vR+xEq3qPQme6Ngeazd8fiMDaUfuwccwhvNUrFl6etpkv4VUCM/t8orZZN3o/+rR41rqudf0wfPXSLvWZnT9oLSrfU9UF70Dv3NeoUFutk7/hmpF70eVv+g5c8tA083YXAHXr1lX/ymUH7sKrYmX1pX/kiXAcbFEXnmV84P/MYJttKo+aiCsH/4sf2jbCkV7tUW3SeyhWwV992Qe8Pg1H/vEofghtjF8TFqP6tDiXvZeiqGH1JlgydAsertXitusndP83EnYtQvj0xoj59Em8F/X5LZNf6MRR1wS7ijtmvmLZynizVyz6xYWj7aS6KF3cB0+1ss18ZPBAFPcqgbCpD6Lnuy0xrMME1KxQR62b2WcBhi3oqz6z237cjFGdJrvoneib+7f/EY+VOz5Vf8OoWWF4LeIdVCtfEzry1DTzdhcAU6ZMUdcjDhkyRC3LpQm682n9KC793xZcO3lcLZ/9NF6d1d/s5Jgh+GXq6+pn76rVYcm+BsuVK/IHwPGRA5F9Jl2t+33fThSvwRaAmz3Z8nnMXDMOe46l3HZ99JwuWLR1tvWs4Nffz2nd3+zIO4O5gjtmPqTeo9h+ZAtOnb+R+UVb49G5iW3mPT08UaaEj2oZKFG8lAyCQHbONVWcyjrfUuXUdrLNlWu/u+R96Jz7pf83Dwm7Plc/p184hXO/ZaJKuerQkYemmbe7ANi+fbu6NaFMOyjzFLdp0wbbtm2DzryrVLN++Ytrp07AO+A2H8ScHNSatxJ//XIrMj+Jw/WLWer/Za3/6sZ6Ly9UeWUSfv1qmRP3vuh7+fMB2HTg6zzXy5e9PKRJNe6ZFYhNfBs60/VswEyZr3JPNfzyvy9/8cuvJxDwpy+fBVti1Zf8tgkn8d3YI/h82xycPJ+mPqsvfzEAnz6fiG3jT6B7YBTeXz/JBe9C79wv2/4JLv+vcOrVrJ9qhdmftsOJe2gcXTNvdwEwdepUzJ8/H76+vmpSELn38KRJmn/4bzO3suX69dtuevSpCHzfpBru6dhdjRuwvkRZX9SenwDLtas4PeU1h+6uu+o4vYlqjh3eYaJqPtSVrv2BZsq8h8etmb9usc18TPgEpJ7ah2bjAhA8oYbqx+7QqDsq+lbB2IgZ6PJOIJqPr4Z/rh2P2f1XOXHv3UtUyAuqC2XgnAhcyb4CHXlomnm7CwBpCpTZh3JJv6DcilBn106mwbtygHXZu0pVXDv1x9lBbjdBMf8b06HmnMlAVuKXKPXAjdnQvKvVwF9XJ+Nq2lFVIFiuXXPyO9CbHFRLeN+4heWxzMP4vyPfoUFAI+hK1+ZAM2X+1Pk0VPL9I/OVfavatAiI0IadsWTbR8i5noOzl86ocSrBf22LprVDcDj9IA6e2q+2k+6r2hXronyZG3dMpIIb1+1dPBUyGD3/HYLvT+6Brjw0zbzdBUCtWrWwfPlydVD4+eefVf9gnTo3Bsbo6uKmdSjTvBW8q98YgFL+if7IWmd7p7NyEY+j0guj1M/F7imHsu3CcTF5EzxKlUKdxYk498U8nBj9nOomIPv7Ch9v3l/9LM2wjWs2x46jydCVrs2BZsp8Uuo6NK3Tyjro7PGg/lj/X9vM7z++Ex0b97JeEdC6fgfVn516ci8aVGusrl4Rbep3wOkLJ3HuUqYL3om+RnZ8C41qBKLbP4Pw85kbU03rylPTzNs9D8CECRPwwQcfqBsPxMTEIDAwEG+++SZ0JgP4TowahFrzE+BZvAR+27UNGbHT4Rf1rGoNOD11HE5NHInq0+PVJYAyGOhM/Lu4lLwZ/s/GoHjN2ijX7Qn1EDI24Keuj7j6bRVpMR0mIP3CSXyaPAujF/XH5MfjERk0ADmWHIxfNgRHMn6Aropy4AvDHTN/5mI6Xl08CLOjE9RI/90/b0P8xunqUr9KvlXV4LU3VsTgjZ4fIPHlA8i+no01e5epfmsxcfmLmBOdgOvXc3Dh9/MYOLerq9+SVrmXv+XAdiNVS8yiwZus6+Vv/p8f/1jWhaemmbf7XgCXL1/Gl19+iR49euD06dNYsWIF+vXrhxIlShR6J3gvAOPxXgCuuxdAVPf8P8/zl+l1tHBE5gXvBUDukvsoTTNvdxfAiBEjcOzYMfVzmTJlcOnSJXV3MCLSuz8wL8w8kXtm3u4CIC0tTTUDCh8fH3Vv4tyDAxHp2x+YF2aeyD0zb3cBIP2AO3futC7v3bsXJUveGMFNRPpeEpQXZp7IPTNv9yDA8ePHq9uC/vbbb2q5bNmyalQwEd1QlCv+wmDmidwz83YXAA8++CBWr16Nc+fOwcvLSx0MiEj/g0FemHki98y83QXAjh07MHv2bHU2IBcQyIQgJ0+exIYNGxyzh0SkyLX48fHx6np8GYTXvn17m/XyJS2z9Ek2o6Oj0avXjWvY7xYzT+Seubd7DMCYMWPQtWtXeHp6on///mpCkHbt2tn7MkRuyxEDguTyu1mzZuGLL77AwoULMW3aNFy8eNG6/siRI5g5cyY++ugjLFmyBO+//746YzcCM0/kmkGAjs693S0AchlQWFgYfvzxR3VAmDhxIrp3727vyxC5rTsFPjQ0NN/1iYmJtzyXnJyMkJAQNQpfyGQ8SUlJCA8Pt/6fbt26oVy5G3eok7n7S5cuDSMw80TOz7wzcm93C4C8uIwClvnAN2/erKqRrKwse1+GyG054ppguQuf3Ignl7+/PzIyMmwu1ZPr8+WOfREREWrU/t1O1JOLmSdyzTwAjs693S0A8uKrVq1So4KXLl2qmgIHDhxo78sQmfZsIK9qPz/S9y73obf5PTfdxTI7OxspKSn4+OOPce3aNfTu3RuNGjUyZM5+Zp7I+Zl3Ru7tLgDOnj2L2NhYFCtWTP1LRLYccd2vnAWkpqZalzMzM9GwYUObMwNpKswdof/www+r7Y0oAJh5ovw56lp/R+fe7i6A4sWLqzOAp59+GoMGDbI+iMhxzYHBwcGq70+a3s+fP6+qfukPzPXII4+o5vnff/9dNQnu3r1bNdkbgZknck0XgKNz71WYecGJyLnXBAcEBKhLfKSJT5r9Bg8eDD8/P9XvFxcXpyr/nj17qoes79u3L+69915DfjczT+SaeQAcnXu77wboCLwboPF4N0DX3Q1wRIf8P88z1rg8ckUC7wZI7pL7EZpm3u4WACLKX1Ge+5uIjOehaeZZABAZjOe1RObiAT2xACAymK7zghORuTLPAoDIYLo2BxKRuTLPAoDIYLqeDRCRuTLPAoDIYLoeDIjIXJm3eyIgIiIi0h9bAIgMpuvZABGZK/MsAIgMpuvBgIjMlXkWAEQG0/WaYCIyV+ZZABAZTNezASIyV+ZZABAZTNdrgonIXJlnAUBkMF2bA4nIXJlnAUBkMF2bA4nIXJlnAUBkMF0PBkRkrsyzACAymK79gURkrsyzACDSpD9w+fLliI+PR05ODkaOHIn27dvbrI+KisK5c+fg6Xljgs+5c+eiQoUKDtobInLGGABH5p4FAJEGzYGnT5/GrFmzsGTJEly9ehWRkZEICgqCj4+PWm+xWHDixAmsX78eHh66Dkki0pOng1oAHJ173guAyAHNgfk9CiM5ORkhISEq+H5+fggMDERSUpJ1/eHDh9UZwlNPPYVu3bph3bp1xr0hInJ65p2Re7YAEDn5bCA0NDTf9YmJibc8l56ejkqVKlmX/f39kZGRYV2+cOGCOjMYP368ag7s06cPGjRogBo1ahTmLRCRizPvjNwXiQJgRX1X7wFR0W4OlKa+Pzfx5fb5iSZNmqiHCAgIQLt27bB161YWAEQadwFYHJz7IlEAEJlJXtV+fuQsIDU11bqcmZmJhg0bWpd37NiBa9euqbOBXF5ejDeRrpl3Ru45BoDIAWcD+T0KIzg4WPX9ZWVl4fz580hJSVH9gbkuXryIGTNmqIFCZ8+exaZNm9CiRQvj3hQROTXzzsg9TxGINGgOlOa96Oho9O7dG9nZ2Rg8eLAaFBQREYG4uDi0bt0aO3fuRNeuXXH9+nUMHz4clStXNn5HiMhpXQCOzr2HRToZXGxiG162ZLS5N7qFyCBHZxY8JrHN8v88P5fi8sgVCbVimHtyj9zHapp5tgAQGUzXaUGJyFyZZwFAZDBdpwUlInNlngUAkcHYsE1kLh7QEwsAIoPp2hxIRObKPAsAIoPpejAgInNlngUAkcF07Q8kInNlngUAkcF07Q8kInNlngUAkcF0bQ4kInNlngUAkcF0bQ4kInNlngUAkcF0PRsgInNlngUAkcF0PRgQkbkyz7sBEhERmRBbAIgMpuvZABGZK/MsAIgMpuvBgIjMlXkWAEQG0/WaYCIyV+ZZABAZTNezASIyV+ZZABAZTNdrgonIXJnnVQBEDmgOzO9RWMuXL0fHjh0RFhaG9evX57nd0KFDMWfOnLv4TURUFDLv6NyzBYBIg+bA06dPY9asWViyZAmuXr2KyMhIBAUFwcfHx2a7VatWYevWrXjooYeM3wkicmoXgKNzzwKAyMkHg9DQ0HzXJyYm3vJccnIyQkJCrMEPDAxEUlISwsPDbQ4WixYtQu/evQu760RURDLvjNyzC4DIAf2B+T0KIz09HZUqVbIu+/v7IyMjw2abcePGYcyYMfD29r7bt0BELs68M3Jf4BaAXr164fLly3muT0hIsPuXE7mjO/X55VXt58discDDw/aVPT3/qN8XLlyIBg0a4IEHHsDGjRthBGaeyHWZd0buC1wAvPXWWxgwYACmT5+OqlWr2v2LiMzCEf2BchaQmppqXc7MzETDhg2ty+vWrcOZM2ewYcMG9a8cJHx9fdWXeGEx80SuHQPg6NwXuACoW7cuRo8ejY8++ggffPCBve+DyDQccUlQcHAw4uPjkZWVhZycHKSkpGD48OHW9ZLLXP/+979RunTpu/ryF8w8kWsvA3R07u0aBCiXIsiDiJx7NhAQEIDo6Gg10Cc7OxuDBw+Gn58fIiIiEBcXh8qVKxv/S5l5Ipe2ADg69x4W6WSwgzRH1K9f3/qvESa20XUixaJrbhNX74F7OTqz4DHZUCf/z3O7w3rNGuKIzItaMcw9uUfuN2iaebuvAnjllVds/iUi98bME7mnQs8DYGfDAZFp6Dov+J0w80TulXlOBERkMF0PBkRkrsxzIqD/qRcSgUEf7cMLCw6h4/BYeBazrY2Kly6LV9ZcwsDZu6yP8lXr2GzTvOeLiJ71f07ecz14F/PGgue+Qbv7O92yrnyZCpgTnYCvR+5Rj7AHu0FnjpwXnIzz9wcisHbUPmwccwhv9YqFl6dt5kt4lcDMPp+obdaN3o8+LZ61rmtdPwxfvbQLX7+0G/MHrUXle3iZpL25zxVStz1WDNsKnXlomnnPwlwaJOrVqwd3UcavMjrGxOKzUeF4v29dFC/lg6bdBttsU71BcxzesR5x0U2sj3MnD1vXV63fFC16j3LB3hd9Das3wZKhW/BwrRa3Xf9C+zH4KT0V4dMeQvTszpjxxDyU9C4Fnc8G8nvoxh0zX7FsZbzZKxb94sLRdlJdlC7ug6da2WY+MngginuVQNjUB9Hz3ZYY1mECala4UfTP7LMAwxb0Rfj0xtj242aM6jTZRe9E39xLcfBi2Ot4L+pzeHoWg848Nc283QXAlClT1PWIQ4YMUctyaYLu6jz8KNL2b8GFjONqedfqeDwQGmmzTfWGwfDxq4Jn3k9WZ/n1W/1xllrC5x5VQCTGvez0fdfBky2fx8w147DnWMpt13t6eMKnpK/6uVTxMriafQU6c9S0oK7ijpkPqfcoth/ZglPnb2R+0dZ4dG4SecvnskwJH9UyUKJ4KRkEgeyca2pmNlnnW6qc2k62uXLtd5e8D51z37ROK/iVqYhRi56B7jw0zbzdBcD27dvVdcFRUVFqnuI2bdpg27Zt0JlvxWrWL39x4cwJ+FasbrPN9es5SE1ahrkvtMCyN55A+LD3UanOg2pdl5FzkPTJG/g1/ZjT910HL38+AJsOfJ3n+n9/8yaC7m2DlAkn8dVLuzHtqzG4rPEBVdfmQDNlvso91fDL/778xS+/nkBAOdvML9gSq77kt004ie/GHsHn2+bg5Pk0NRjy5S8G4NPnE7Ft/Al0D4zC++snueBd6J375B82YNyyIbjw+3nozkPTzNtdAEydOhXz589X0w3KNIVy/+FJk/T+8Ht43PpnsFiu2yx/t2AStnw2Rf189vgP+H7TYtQNfgzNegxFVuZJHPxupdP2191M7P4eVuxYgGbjqqLd5HqqS+Chmk2hK12bA82e+et/ynxM+ASkntqHZuMCEDyhhurH7tCoOyr6VsHYiBno8k4gmo+vhn+uHY/Z/Vc5ce+pqPHUNPN2FwDSFHjz7EPSL3j9um1wdPNrehrK+gVYl8tWqGrTIiCadnsBZStWsy5LM2BO9jU82L4PajVpqwYFdh45GxX/cj+iZm5w6v7rLrRhZyzcGq9+Pn72KDanrkGzOo9AV7o2B5op86fOp6GS7x+Zr+xb1aZFIPdzuWTbR8i5noOzl84gYdciBP+1LZrWDsHh9IM4eGq/2m7R1tmoXbGuGsxK5uShaebtLgBq1aqF5cuXq4PCzz//rPoH69SxHQ2vm8Pb16Fmo1a4p3JNtdykY38cSra901n1+4MR1HOY+lkKgQaP9MDBLSsx57nm+PDpB9WgwIRp0cj4+XvMj2nnkvehq/0ndqLTQzfmr/YteQ+a39s6z35DHeh6NmCmzCelrlN90NXK38j840H9sf6/tpnff3wnOjbuZb0ioHX9DupzmXpyLxpUa4zqfrXUujb1O+D0hZM4dynTBe+EigJPTTNvdwEwYcIE/PDDD+rewzExMeqg8Oabb0Jnl86l48t3BqH3pAQ8Pz9VTu/xn8+n4+Euz6LN0xPUNmvfG4by1f6qLhXsO/0brPtguOoKoMKJ6TDBelnV8E+j0LZBR3Wp1RdDkvDxt/9CyuEk6ErX/kAzZf7MxXS8ungQZkcnIPEVuduaB+I3TlefSflsijdWxKBKuepIfPkAVg3fjm0/fYtl2z/B4YxDmLj8xRuXrr60G8+FvoyBc7u6+i1pl3t34qFp5u2+F4DcH/zLL79Ejx49cPr0aaxYsQL9+vVDiRIlCr0TvBeA8XgvANfdC2Bn1fw/z387WYRPCZyUecF7AZC75H6nppm3uwVgxIgROHbsxmj3MmXK4NKlSxg5cqQj9o1IS7o2B+aFmSdyz8zbXQCkpaWpZkDh4+Oj7k2ce3AgIn0PBnlh5oncM/N2FwDSD7hz507r8t69e1GyZEmj94tIW7r2B+aFmSdyz8zbfTOg8ePHq9uC/vbbb2q5bNmyalQwEd1QlC/7KQxmnsg9M293AfDggw9i9erVOHfuHLy8vNTBgIj+UJSb/AqDmSdyz8zbXQDs2LEDs2fPVmcDcgGBTAhy8uRJbNjAyW+I4MAmP7kWPz4+Xl2GJ4Pw2rdvb7N++vTp2LhxIzw9PfHcc8+p6XuNwMwT5c+RzfyOzL3dBcCYMWPUIKBFixapS4ESExNRv359e1+GyG054mxALr+bNWsWlixZgqtXryIyMhJBQUFqUJ7YunUr9u/fj4SEBFy4cEEdBORAUbx48bv+3cw8kWtaAByde7sLALkMKCwsDD/++KOqOCZOnIju3bvb/86ITHowCA0NzXe9fMH+WXJyMkJCQqzBDwwMRFJSEsLDw9WyHBTkOcmk3LBHDgDFihlzi1Vmnsj5mXdG7u2+CqB06dJqFLDMB75582ZcvHgRWVlZ9r4MkdtyxLzgEm65EU8uf39/ZGRk2Gwj/fOTJ09WX849e/Y0rABg5olccy8AR+fe7hYAmf1r1apValTw0qVL0a5dOwwcONDelyEybX9gXtV+fqTvXW5AdTOp+v9Mcin9gE8++aT17OBuMfNEzs+8M3JvdwFw9uxZxMbGqipD/iUix/cHyllAaqrMWX9DZmYmGjZsaF0+cuSI6iOUs/Ry5cqpZsNDhw4ZUgAw80SuGQPg6NzbXQBIH4OcAdx33302c4F/+OGH9r4UkVtyxDXBwcHBaiSwNL3LaOCUlBQ1MC+XzMwXFxeHefPmqbn7t2zZgrffftuQ383ME7lmHgBH596rMPOCE5FzzwYCAgIQHR2N3r17Izs7G4MHD4afnx8iIiLUAaB169Zqtr4uXbqoM/W+ffvi/vvvN+R3M/NErmkBcHTu7b4boCPwboDG490AXXc3wLRy+X+ea5x3eeSKBN4NkNwl92maZt7uFgAics9pQYnIXJlnAUBkMF2nBSUic2WeBQCRwXQ9GBCRuTLPAoDIYOzZJjIXD+iJBQCRwXTtDyQic2WeBQCRwXRtDiQic2WeBQCRwXRtDiQic2WeBQCRwXQ9GyAic2WeBQCRwXQ9GBCRuTLPAoDIYLoOCCIic2WeBQCRwXTtDyQic2WeBQCRwXRtDiQic2WeBQCRwXRtDiQic2WeBQCRwXQ9GyAic2WeBQCRwXTtDyQic2WeBQCRwXRtDiQic2Xe09U7QOSOzYH5PQpr+fLl6NixI8LCwrB+/fpb1k+bNg2PPfYYOnXqhPnz59/dmyAil2fe0blnCwCRBv2Bp0+fxqxZs7BkyRJcvXoVkZGRCAoKgo+Pj1q/adMmHDx4ECtXrsSVK1fQs2dPtGzZEvfee6/xO0NEThkD4OjcswWAyAH9gfk9CiM5ORkhISEq+H5+fggMDERSUpJ1fUBAAGJiYlCsWDGULl0aNWvWVAcPItIz887IPVsAiJzcHxgaGprv+sTExFueS09PR6VKlazL/v7+yMjIsC7Xq1fP+vOePXuwf/9+NGrUyL4dJ6Iik3ln5L5IFACvb9J0BEUR9rqrd8DMLHc8GhTiJS3w8LA9l/D0vLUBb/fu3RgyZIjqF8xtJiyqjs5k7slNWIzPvDNyXyQKACIzyavaz4+cBaSmplqXMzMz0bBhQ5tttmzZgtGjR6uDQHBwsCH7SkSuybwzcs8xAEQakGBL319WVhbOnz+PlJQU1R+Y69ixYxg1ahRiY2P55U/kJoIdnHu2ABBpQAb7REdHo3fv3sjOzsbgwYPVoKCIiAjExcVh9uzZapTw2LFjrf/npZdeQqtWrVy630RUdHPvYZFOBiIiIjIVdgEQERGZEAsAIiIiE2IBQEREZEIsAIiIiEyIBQAREZEJsQAgIiIyIRYAREREJsQCgIiIyIRYABAREZkQC4C7IJMovvnmmwgPD0e3bt1w5MiRQm1Dt94De9CgQbddJ9NeDhs2DB07dkSfPn1w5swZp+8fmRtz7xjMvfOxALgLX3/9tbo3s/z7+uuv28zHbM829MdBc8GCBYiJiVE/3868efPU/NhfffUVIiMjMWXKFKfvJ5kbc28s5t51TFEAnDp1SlWN3bt3xxNPPIGff/4Z69evV8tyU4VXX30V165dwzfffKOq9ZycHGzfvl1Vm5cvX0a/fv3Udjc/5LaMcpemxx57TP2OJk2aqMCfPXvW5ncXZBvdOOrv+csvv2Dv3r2YOHFinr/722+/RefOndXPcnYly7ydBd0Oc28s5t79mOJugEuXLrU2Ha1atQo7d+7Ep59+ik8++QRlypRR1eRnn32Gp556CgkJCfj444+xbNkyTJ48GSVLllTLt5Oeno6KFStalytUqKCek7s12bONbhz19xRTp07Ftm3b8lx/89+zWLFiKFGiBC5dugQfHx+HvFfSF3NvLObe/ZiiAAgKCsLQoUOxb98+tGvXDp6enkhLS1NVbG7/UvPmzdXP0lwnFWaPHj3w0EMPqeekcj137pzNa86dO/e2Fai89s0Kso1uHPX3lINkYXh4eNz1eyL3w9wbi7l3P6YoAAIDA1VFumHDBlWtnjhxAiEhIZgxY4Zaf/HiReu20hwl1eWBAwesz+VVuVaqVMlmMIo0Z91c9Rd0G9046u9ZEPL3zP0bShOjNDnK2QfRnzH3xmLu3Y/eJWkBTZs2TfVV9erVCy+++CK8vLxUc5N8SKVSf+2117B48WJkZ2ern9955x2ULl1aPZcf+fCvXr1avcaePXvg6+uL8uXL272Nbhz19ywI+XuuXLlS/bx27Vo0bdrUgHdE7oi5NxZz735M0QIgfVYjRozAwoUL4e3tjUmTJqnqfMCAAaqafOCBB9C3b1/MmTMH9913H5o1a4aqVauq0aatW7dW1eftSH/Yrl271GAfeV3p6xLSRPavf/0L8fHxeW6jM0f9PfOSmJiozjreeustREVFqcFGnTp1QtmyZdVBhuh2mHtjMffux8PCoZRERESmY4ouACIiIrLFAoCIiMiEWAAQERGZEAsAIiIiE2IBQEREZEIsAIiIiEyIBQAREZEJsQAgIiIyIRYAREREJsQCgIiICObz//Uq3Y62dzdIAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 600x200 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_predicted_roc_pivot = calc_intersectionalbias_matrix(ds_predicted_roc)\n",
    "df_predicted = calc_intersectionalbias_matrix(ds_predicted)\n",
    "plot_intersectionalbias_compare(df_predicted_roc_pivot,\n",
    "                                df_predicted_pivot,\n",
    "                                vmax=0.8, vmin=0, center=1,\n",
    "                                title={\"right\": \"RoC\", \"left\": \"ISF(used RoC)\"})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eb02bdcc",
   "metadata": {},
   "source": [
    "* Compared to `ROC` (the left-hand quadrant), `ISF (ROC is used)` (ROC-leveraged ISF; the right-hand quadrant) achieves a better bias mitigation.  \n",
    "* We have confirmed ISF helps ROC mitigate intersectional bias.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "90fe261e",
   "metadata": {},
   "source": [
    "#### Compare accuracies for `ROC` and `ISF (ROC is used)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9bcf7d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n",
    "df_acc.loc['ROC']=(\n",
    "    accuracy_score(y_true=Y_test, y_pred=ds_predicted_roc.labels),\n",
    "    precision_score(y_true=Y_test, y_pred=ds_predicted_roc.labels),\n",
    "    recall_score(y_true=Y_test, y_pred=ds_predicted_roc.labels),\n",
    "    f1_score(y_true=Y_test, y_pred=ds_predicted_roc.labels)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b4dc8958",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1 score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LR Model</th>\n",
       "      <td>0.848824</td>\n",
       "      <td>0.744169</td>\n",
       "      <td>0.607855</td>\n",
       "      <td>0.669140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ISF (ROC is used)</th>\n",
       "      <td>0.830692</td>\n",
       "      <td>0.693913</td>\n",
       "      <td>0.584701</td>\n",
       "      <td>0.634643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ROC</th>\n",
       "      <td>0.805852</td>\n",
       "      <td>0.578396</td>\n",
       "      <td>0.841149</td>\n",
       "      <td>0.685455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Accuracy  Precision    Recall  F1 score\n",
       "LR Model           0.848824   0.744169  0.607855  0.669140\n",
       "ISF (ROC is used)  0.830692   0.693913  0.584701  0.634643\n",
       "ROC                0.805852   0.578396  0.841149  0.685455"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_acc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ec347540",
   "metadata": {},
   "source": [
    "* Accuracies for `ISF (ROC is used)` and `ROC` are almost the same.  \n",
    "* This indicates extending RoC with ISF does not cause significant accuracy degradation.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (venv3.11.7)",
   "language": "python",
   "name": "your-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
